XT- 6-95 FRI 9:43 


NASA— CR - 199440 


P. 02 


The CCD/Transit Instrument (CTI) date-analysis system 

N. G. H. Cawson, J. T. McGrow, M. J. Keane 

Steward Observatory, University of Arizona 
Tucson, Arizona 85721 

Abstract 


sVJztv- 3* y 

/O 


The automated software system for archiving, analysing and interrogating data from the 
CCD/Transit Instrument (CTI) is described. 

The CTI collects up to 450 Mbytes of image-data each clear night in the form of a narrow 
.trip oi sky observed in two colors. The large data-volumes and the scientific aims of the 
5 2f oject make it imperative that the data are analysed within the 24-hour period following 
jt' the observations, to this end a fully automatic and self-evaluating software system has 
jcen developed. 

The data are collected from the telescope in real-time and then transported to Tucson x 
\ toe analysis. Verification is performed by visual inspection of random sub-sets of the data 
jnd obvious cosmic rays are detected and removed before permanent archival is made to 
optical disc. 

The analysis phase is performed by a pair of linked algorithms, one operating on the 
t obsolete pixel-values and the other on the spatial derivative of the data. In this way both 
f isolated and merged images are reliably detected in a single pass, jn order to isolate the 
v latter algorithm from the effects of noise spikes a 3x3 Hanning filter is applied to the 
row data before the analysis is run. The algorithms reduce the input pixel-data to a 
database of measured parameters for each image which' has been found, 

A contrast filter is applied in order to assign a detection-probability to each image 
and then x-y calibration and intensity calibration are performed using known reference 
stars in the strip (e.g. AGK-3 stars). These are added to as necessary by secondary 
standards boot-strapped from the CTI data itself. 

The final stages involve merging the new data into the CTI Master-li6t and History-list 
and the automatic comparison of each new detection with a 6et of pre-defined templates in 
parameter-space to find interesting objects such as Supernovae, quasars and variable stars. 

Each stage of the processing from verification to interesting image selection is 
performed under a data-logging system which both controls the pipe-lining of data through 
% the system and records key performance monitor parameters which are built into the 
software. Furthermore, the data from each stage are stored in databases to facilitate 
♦valuation, and all stages offer the facility to enter keyword-indexed free-format text 
into the data-logging system. In this way a large measure of certification is built into 
the system to provide the necessary confidence in the end results, 

1 . Introduction 


The CCD/Transit instrument (CTI) data analysis system has to be capable of handling up 
10 450 Mbytes of image data within the 24 hour* following each observation being taken. Por 
*uch a data handling problem novel techniques have had to be devised, the most fruitful of 
*hjch ha$ been the use of a database approach to all aspects of the project Including data 
^presentation and storage, data manipulation, and data retrieval and interrogation (see the 
accompanying paper by Cawson ejt al. ) . 

Figure 1 shows a block diagram of the CTI daily data analysis system. The analysis 
I system is in the form of a pipe-line through which all the data pass sequentially and this 

* t '* PBr is organised in a similar fashion. 

. Section 2 gives a description of the control system which holds the various stages of 
processing into a pipe-line capable of being operated by non-experts. Section 3 describes 
i * va ri©us stages in the analysis pipe-line, in particular the actual extraction of 
«hi K #ation from the pixel data in Section 3.6, the definition of the CTI knowledge base 

* ia produced by the system in Section 3.9 and the extraction of astronomical 

r«wi n<,t lon the knowledge base in Section 3.11. Section 4 covers the archival of the 

-? el data and the extraction of 6ub-sots of data from the archive. Section 5 
ot the coaddition of pixel data from many observations and the subsequent analysis 

tn e coadded data. Conclusions are drawn in Section 6, references quoted in Section 7 and 
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aeKnowledgansente of support given in section 8. 

2, Analysis control and logging system 

All observational data from the CTi telescope pees through the analysis pipe-line 
sequentially under the control of a Daily Analysis Logging system. This control system is 
actually a database (known as the 'DAL-file') with one entry for each observation made with 
t he CTI telescope. As each observation passes through the data reduction system the entry 
in the database is updated and key software performance diagnostics are also entered into 
the DAL-file to monitor the stability of the reduction system to data taken under widely 
differing observing conditions (e.g. phase of the moon, seeing, galactic latitude, time of 
night etc,) . The DAL-file also prevents accidental operator error as one stage of the 
reduction cannot be initialised unless the DAL-file 6hows that the necessary previous 
stage® have been completed successfully. Similarly, a stage cannot be repeated 
accidentally. 

Because the DAL-file is a database it may be interrogated in the same manner as any 
other database, however, in this case the types of questions that are answered by the data 
jn the DAL-file are of the form: "How far has a particular observation passed through the 
6ystero?“, "What is the next observation to be processed by a particular stage?"} "what was 
the mean image (or cosmic ray) density detected in a specified observation?"; "How stable 
ar e the geometric calibration coefficients over different time scales?" etc. As data are 
collected over longer periods of time so the DAL-file will also be useful for answering 
questions on the percentages of night time spent observing in different Colours or not 
observing due to cloud etc. 

3 . Desc r iption of t he CTI dally data analysis syste m 
i Ent erin g observations i nto the syste m 

Each clear night between 200 .and 450 Mbytes of pixel data are transported from, the 
telescope on Kitt Peak Mountain to Steward Observatory on magnetic tapes. The 
characteristics of these data are given in the accompanying paper by McGraw et al, 

The first stage of the data analysis system involves entering the observations into the 
system. The data tapes from the telescope are mounted on the CTI Data General MV/10000 
computer and a brief inspection of the data at the beginning of each tape is made using the 
IIS image display hardware. The UT at the start of the observation ia read from the tape 
header and the positions of a few AGK-3 3tars are marked with the Cursor to achieve 
approximate geometric calibration. 

The system then takes over automatic control and the entire observation is copied onto 
magnetic disk for further processing. As data around each reference star are read from 
tape the positions of the stars are measured automatically by an algorithm which centres on 
the steepest gradients in the pixel data. This is done to avoid using pixels at the image 
centre itself which are likely to be highly saturated. Image centres are measured to an 
accuracy of les6 than quarter of an arc-second and the rms residuals from using all AGK-3 
stars observed (typically over one hundred) is less than one arc-second (more accurate 

calibration is performed using thousands of secondary standards at a subsequent stage 

Bee balow) , 

Several automatic tests are performed on the data ae they are copied from tape to disk 
and assuming the data passes these tests a new entry is made into the DAL-file for the 
observation. This entry includes the date, colour and duration of the observation and its 
approximate geometric calibration. 

lei _Data ver i fic ation 

The second stage of processing is the formal verification of the data by visual 
inspection Of sub-set6 of the data at a number of randomly chosen locations. Visual 
inspection has proved to be the most efficient way to verify: 

U) that the flat-fielding has been correctly applied resulting in image data free from 
Structure across the field (background variations, linear gradients and Simple 
curvature are permitted and can be caused by diffuse light from the moon or the sun 
near twilight) ; 

(2) that the structure of images in the data is good (sharp point spread function with 
no elongation caused by incorrect dewar rotation etc.); 

<3) that the mean sky background values are reasonable (at different angles from the 

moon it is necessary to change filters to keep the photon noise comparable with the 

* 
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ttl , sensitivity of the CCD chip to cosmic rays. 
j,4 _ Analysis of the pixel data 

The single most important stage in the analysis of the observations Is the reliable 
detection and parameterisation of each astronomical Image represented in the pixel data. 
th i6 amounts to a transformation of information from pixel representation to attributes of 
,he images stored in a database, a pair of inter-linked algorithms operating on the pixel 
data (after the fitted background estimates have been subtracted from them) are used for 
this transformation. 


In the first algorithm, all pixels 
below a pre-set isophotal threshold 
are ignored. Those above the 
threshold form groups which are 
connected either adjacently or 
diagonally. All pixels in each group 
are considered to 'be part of the same 
•image*. The properties of the 
•image' are calculated from the first 
and second moments of the group of 
pixels of which it i3 composed. These 
moments provide the integrated 
luminosity, centroid, radius of 
gyration, ellipticity amd position 
angle of the image. The peak pixel 
value and its position and the local 
background are also stored for each 
image . 

In the Second algorithm, all 
pixels are allowed to migrate to the 
highest neighbouring pixel (adjacent 
or diagonal) and from there upwards 
until a peak is. reached. In this 
manner the pixel data is segmented 
into single pixels at the location of 
each peak and separated by valleys and 
saddle points in the data. During £he 
migration, first and second moments 
are summed for all pixels which arrive 
at each peak. This results in 
'segments' for which the same 
parameters are available as for 
' images' . 

The two algorithms are inter-linked by a third which cross-references the segments and 
images. If any pixel is common to both an image and a segment, the segment is made to refer 
to the Image and the image updates a list of Included segments (the definition of a segment 
“akes it impossible for a segment to refer to more than one image). Segments may have a 
’parent' image and images may have several segments, however segments do not necessarily 
have a parent image if all pixels in the segment are below the isophotal threshold for 
images . 

The three algorithms have been designed in such a way that they can be executed together 
lri » single linear pass through the data with each pixel being considered only once. 

There are two principal advantages of 'segments' over ' images' t no threshold is required 
lo segment the image data so the luminosity contribution to an image below the isophotal 
threshold is not lost; and blends are detected as such so long as each object has its own 
“ l3 tlnct peak. The only disadvantage of segments is that every peak in the data is 
guaranteed to produce a segment and noise ensures that about one pixel in nine all over the 
ackground is a peak (if adjacent connectivity only were allowed then about one pixel in 
*»ve would be a peak, which accounts for the use of diagonal connectivity in the 
"iaotithms). 

However, in practise, segments and images are thrown away if their integrated value over 
thr ^5 Xele (their luminosity) is below some detection threshold. Furthermore, an isophotal 
f r e6 ho\d of aero above background is usually used for segments to prevent their properties 
the? be ' ,n 9 biased by a 'moat' of negative luminosities around each segment migrating to 
r [>eaks along with the central positive pixels. 
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figure 3. Segmentation of an extended image. 
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These steps virtually eliminate 'background segments' but extended objeats such as 
galaxies are also segmented at a radius where the gradient ©t their profiles becomes less 
than the pi^el-to-plxel noise inherent in the data (see figure 3). It has been found that 
a simple 3x3 Banning smooth of the data prior to analysis reduces such segmentation and 
allows more concise separation of noise from 'real' segments at the next stage. The 
smoothing acts much like a matched filter at improving detection reliability and is 
performed aa the data are read into the analysis program without editing the raw pixel fij t 
on disk. 

3 . 7 Selecti on of 'real' segm e nts 

AS indicated in the previous section, the segmentation algorithm can produce spurious 
detections due to local peaks in the data caused by noise. In order to remove such effects 
and to assign a 'probability value' to each detection the concept of a contrast filter is 
used. 

when visually inspecting the image of a bright galaxy the eye is able to distinguish 
undulations in the surface brightness of the galaxy due to large-scale structure, such aa 
spiral arms, from superposed Images like stars or H-II regions. It seems clear that this 
ability Is based on a comparison of the 'peakedness' of the region in question with a 
measure of the mean surface brightness of the surrounding environment. In othor word3 the 
eye picks out regions centered on peaks on the basis of contrast from their surroundings. 

In an exactly parallel manner segments may be plotted on a scattergram with peak surface 
brightness plotted against mean surface brightness (i.e, total luminosity divided by area 
or number of pixels in the segment). Such a scattergram is shown in figure 4a, Clearly no 
segments can lie below the y=x line (eero contrast) but segment* caused by local noise 
spikes in the data superposed on an enhanced galaxy background appear very Close to the 
line and increasingly so at higher surface brightnesses. This suggests rotating the axes 
Of the plot through 45 degrees and using an exponential asymptote to filter eegments close 
to the 'zero contrast' line. Figure 4b shows the result of this transformation where 
circles have been used to represent images which were selected by careful inspection by 
eye. A logarithmic scale hae been used for the y-axis so the line represents the desired 
exponential asymptote. 
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figure 4. Contrast filter to select 'real' detections, 
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The line rather successfully divides 'noise' segments below from 'real' segments above. If 

we arbitarily define the 'probability-value* of segments below the line as aero then the 

distance of a segment above the lina can represent a measure of our confidence that the 
segment is a 'real' detection. We call this measure the 'probability-value' (or p-value) of 
the segment because It monotonically increases with genuine detection probability although 
its zero point is arbitarily defined and it is not normalized. However we may choose to 
consider as 'definite' detections all segments which have a p-value above some threshold. 
Furthermore, if we detect a segment at the same x-y position on different nights wa may 
combine the p-values as though they were genuine probabilities to decide whether the double 
uncertain detection warrents the status of a 'definite' detection. Thus the system 
maintains the ability to detect obvious changes within a single day whilst requiring 

repeated confirmations for less certain objects (this capability is referred to as 'fursy 

logic 1 ) . 

3 .8 accur a te p ho tome t ric a nd geometric calibration 

Having thrown away all segments whose properties indicate that they are likely to be 
caused by noiae, the remaining segments are then calibrated accurately in position and 
brightness. 

Luminosity calibration Is performed using photometric secondary standards within the 
field of view of the telescope which were obtained from a series of photometric 
observations with the Steward 1.55ra telescope on Mount Bigalow, 

Initially, for the positional calibration, AGK-3 stars ere used but beeauae most AGK-3 
stars are heavily saturated in the CTI data their centroids are not sufficiently reliable 
(especially as charge bleeding in the CCD occurs under such conditions of saturation). 
Nevertheless, the AGK-3 stars over many nights' observations provide a statistical 
calibration from which many thousands of unsaturated secondary positional standards may be 
derived. These same stars are being calibrated .photometrically using CTI data itself from 
observations made when self-consistency between the photometric secondary standards was 
extremely good. 

All coordinates are calibrated to the 1987.5 epoch (J2000 system) because such a date is 
in the range of the initial period of operation of the CTI telescope. Although 
transformations to this system are rigourously calculated for each object during the 
calibration 6tage, the dominant terms to convert any single observation to thi6 epoch are 
linear so that good approximations may be obtained in a very simple manner when extreme 
positional accuracy is not essentia), 

3.9 Defini tion of t he CTI know l edge base 

The end result of the analysis of each nlght'6 observations is a calibrated li6t of all 
objects which were observed. These lists are then used to create a knowledge base which 
Summarise!* the total information extracted from all the observations to date. This 
knowledge base is maintained in the form of a pair of databases — the Master List and the 
History Li6t. 

The Master List contains the time-averaged properties of each object ever definitely 
detected with the CTI telescope, plus various flags and pointers. The Master List is kept 
sorted strictly by right ascension and in detail each object has an entry of the forms 

• right ascension and declination of the object; 

• integrated luminosity in each of 5 bands (U, B, v, R and I) and two polarisations; 

• the number of observations used to form the 7 average integrated luminosities; 

• the area, radius of gyration, ellipticity, position angle and peak surface brightness of 
the object as measured from the observations of best seeing; 

• the best seeing yet encountered during observations of the object; 

• flags to indicate that the object is a reference star (for position and luminosity 
calibration), an extended object (galaxy), a variable object or an object in a confused 
region (le. sometimes blended with another object); 

• a pointer into an identification list if the object has a name or other information 
referring to it; 

• a pointer into the History List. 

The History List records each individual measurement of the v-magnitude of each object 
sver detected with the CTI telescope and is stored in the order that each object was first 
Getected. In detail the History List has on entry for each object of the form: 

• right ascension and deolination of the object (to provide easy access into the Master 
List); 

integrated V-magnitudes each time the object was detected; 
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• dates on which each detection was made (to enable a light curve to be drawn). 

The user interface to the CTI knowledge ba3e is by means of a powerful database 
interrogation program with graphical I/O capability. Users may extract all objects Obeying 
complex sceintific, relational and/or logical teats on their properties and these tests may 
optionally be specified in graphical or functional form (see the accompanying paper by Y 
Cawson ejt a 1 ■ ) , 

3.10 Mergin g new observations into the Cxi knowledge base 

In order to create and maintain the CTI knowledge base new obsorvations are merged Into 
the Master and History Lists on a daily basis, This stage is conceptually extremely simpl* 
consisting of a search through the current Lists for each object just detected and either 
updating the entries or inserting a new one (the Master List is sorted by Ra and 
observations are made in that order so very little searching is required). However, the 
problem is complicated dramatically by the possibility that two distinct detections on a 
night of good seeing may be detected as a single blended object on a night of inferior 
seeing. For this reason unmatched detections must first be compared with possible blends of 
two or more objects before a new entry is made. Fortunately, the nature of the segmentation 
algorithm makes It extremely unlikely that complex blends of many detections will be needed 
to match a single detection from another night (this would not be true foe the isophotal 
threshold algorithm alone), Searching for blends is therefore limited to individuals, pairs 
or triples from either list matching one another. 

A second problem results from the fact that the telescope appears to have very small 
time-variations in its optical axis. These variations amount to a few arc-seconds over 
time-scales of a few minutes and are almost certainly caused by flexure of the telescope 
due to horizontal thermal gradients across the structure. Because of this flexure it is 
necessary to allow for small variable x-y offsets between new observations and the Master 
List. These offsets are locked to low values (around zero) by the previous calibration 
technique. However, a log of the x-y offsets which are found to be necessary to pull the 
positions of objects into exact conicldence as a function of time is put into a database 
for engineering analysis and for use when aligning multiple files of pixel data for 
coaddition (see Section 5.1 below). 

There is one futther complication in the maintenance of the CTI Master and History Lists 
which concerns the threshold at which detections are placed into the Lists. In any noisy 
system with a sharp decision threshold there will be objects which pass the threshold due 
to noise adding to their 'true' properties and similarly other objects which are wrongly 
rejected. Statistically these effects may cancel one another but that Is not sufficient 
when two lists are to be compared and matched On an object-by-object basis. Objects which 
fall close to the threshold may actually fail just below the threshold in one list (i.e. be 

absent from the list) and above in the other list -- the result is that a new object is 

found or an existing one lost and this may trigger some unwarrented action (for instance 
alerting other astronomers). 

To overcome this problem it is necessary to use a 'fuzzy' threshold consisting of two 
thresholds spaced sufficiently far apart that an object crossing from below the lower 
threshold to above the higher one would be sufficiently remarkable to warrent triggering 
the action. This is achieved by storing all objects between the two thresholds in a 
'pool 1 , in practise each object In the pool has a detection probability <p-value) so that 
objects which continuously fall between the thresholds may reinforce one another and 
migrate towards and ultimately cross the upper threshold for inclusion in the Master and 

History Lists. This scheme remains alert to new objects of sufficient certainty of 

detection in a single observation while using the results of several observations tor 
fainter objects. 

3.11 Automatic execution of key projects 

After the CTI Master and History Lists have been updated with the results of a night's 
observations, the observations are compared with the Master and History Lists to search for 
evidence of newly detected interesting objects. For each current CTI project whoea goal i* 
the detection of objects of a particular type, the first step is to interrogate the CTI 
knowledge. base interactively in order to define and refine the logical tests which are 
optimal in selecting the objects of interest and descriminating against unwanted ones. Once 
this has been done, the definitions of the logical tests can be entered as ‘macros' for 
automatic execution on a daily basis. In this way key projects are performed on all the 
data which the telescope collects and 'search lists' of selected objects are automatically 
compiled for each project. 

There is a parallel effort to use AI techniques to analyze and search the CTI knowledge 
base with the aim of producing an expert system to classify objects and to find unusual 

a 


86 / $PI£ Vot. 62? Instrumentation in Astronomy VI ( 1986 1 


I 




one* (see the accompanying paper by Bernat et al ■ ) . 


P. 10 


1 4. ft.rcM.val of the pixel data 

4 • 1 The CTX archival system on optical disks 

Each clear night can produce as much as 450 Mbytes of pixel data so it is imperative to 
jlush this data from active computer memory (magnetic dish etc.) as quickly as possible. 
After the analysts stage the pixel data are not explicitly required by any further stages 
although purging is usually initialised only after the analyzed databases have been 
calibrated and successfully merged with the Master and History Lists. This is so that 
accurate calibration is available for the data when it is archived, so that any problems 
, encountered may be investigated and corrected immediately, and so that raw pixel data is 
immediately available for regions around any newly detected objects of interest. 

when these requirements have been met the pixel data is permanently archived on optical 
disk. This medium offers a compact, random-access archival system at low cost and is the 
subject of an accompanying paper (Keane al_. ) . Approximately two weeks of observations 
can be stored on a single optical disk so^phyeical volume becomes an added advantage over a 
tape-based archival system. 

f Due to the large data volume and the fact that astronomical images are characterised by 
a majority of blank sky background of approximately constant pixel values, there is both 
the incentive and the opportunity to compact the pixel data without any loss of 
information. The scheme which is used involves storing the first pixel as its absolute 
value (using 2 bytes) and then switching to one-byte differences unless two adjacent pixels 
differ in value by more than 121. When this happens the special value -128 is inserted 
into the archive and the scheme repeats. This data compression is referred to aa 'byte 
compaction' and yields a typical data volume reduction of 45 percent. 

, * 

The disadvantage is that it is no longer possible to jump Into the pixel data randomly 
at any location. In fact a region of pixel data near the end of a strip 8 or 10 hours long 
could only be regenerated by unpacking the data sequentially from its start. This problem 
is overcome by re-starting the byte compaction at pre-defined intervals in the data (e.g. 
every 5i2 lines) and recording in a directory the location in the archive of the first 
pixel of each interval. The same directory is used to store the calibration coefficients 
for the observation and all other information which might be relevent when data is to be 
> extracted from the archive (for example the seeing and mean background values, date and 

Cilteri etc.). A database is used for the directory to ease the interrogation of the pixel 
1 archive and to allow the archival process itself to be monitored, for instance to check the 
efficiency of the byte compaction algorithm. 

4_._2 ext ra ctio n of image data f rom tha arc hi ve 
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There are many situations which may be relevent when a user wishes to extract data from 
the pixel archive. For example he may have a list of coordinates end want a small region of 
image data around each one in a specified colour, or he may want all observations of a 
particular named object when the seeing was better than some limit. Thus a general-purpose 
program to interrogate the pixel archive directory is required which may create a list of 
the locations of the desired sub-set of pixel data for subsequent extraction in batch mode. 
At present the uaoc specifies exactly which data he requires but a command language driven 
system Is planned where requirements may be entered and the system will Select which data 
is suitable to be extracted. 


5 . Coaddition of the pixel data 
Creat ion of coadded pixel d ata 

Periodically, all the observations taken in a given colour band arc extracted from the 
pixel archive, registered (using the databases of x-y offsets created during the merging of 
the observations into the the CTI knowledge base), and coadded. This procedure produces a 
seamless Image of the entire CTI strip to fainter limiting magnitude than a single 
observation." The coaddcd image is stored in the form of a 4-byte field for each pixel — 
three bytes for the simple sum of the pixels from each observation and the fourth byte for 
the number of values entered into the sum. in this way up to 255 observations can be 
toadded (many years of data) and individual pixels whose values are In any way suspect In 
“ny observation (e.g. cosmic ray pixels) need not be entered into the coadded image. 

During the creation of the coaddcd image all other use of the computer has to be 
Upended and all the magnetic disks cleared for the sole purpose of making room for the 
°added data (1.1 Gbytes per colour). The resulting coadded image is immadiatly archived 
° optical disk at the end of the task and may be 'extracted' in the same manner as nightly 
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5 . 2 an alysis o f the c o added pixe l data 

when the coadded data has reached a sufficiently faint limiting magnitude it will be 
normalised (by dividing the three-byte sum by the number of entries for each pixel) and 
into the input of the CTi data analysis system to create a single list analogous to the 
Master List. This 'Super List' is expected to have approximately 20 million entries to a 
limiting V-magnltude of about 25. 


6. Conclusions 


The CTI telescope has been in part-time operation for just over one year. During this 
time engineering and scientific data has been collected and run through the CTI data 
analysis system. The experiences with this data have resulted in the analysis system 
described in this paper. Although some sections of the analysis system have not yet been 
completed it is unlikely that the overall structure of the 6ystem will change, 

The use of a database approach in all aspects of the project has dramatically eased the 
design of the system and has speeded up Its implementation and the analysis of problems as 
they have been found. So far the entire system ha9 required just two man-years of software 
effort which has included the design and implementation of the database definition and 
interrogation system. 
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